Papers with capacity-gap problem
Pro-KD: Progressive Distillation by Following the Footsteps of the Teacher (2022.coling-1)
Copied to clipboard
| Challenge: | Knowledge distillation (KD) is a powerful tool for deep learning applications. |
| Approach: | They propose a method which defines a smoother training path for the student by following the training footprints of the teacher rather than solely relying on distilling from a single mature fully-trained teacher. |
| Outcome: | The proposed technique is quite effective in mitigating the capacity-gap problem and the checkpoint search problem. |